On the Accuracy and Completeness of the Record Matching Process

The role of data resources in today's business environment is multi-faceted. Primarily, they support the operational needs of an organization or a company. Secondarily, they can be used for decision support and management. The quality of the data, used to support the operational needs, is usually below the quality required for decision support and management. Recent advances in information systems investigate ways to estimate, improve and preserve the quality of data, so as to increase their value in both domains. Record matching or linking is one of the phases of the data quality improvement process, in which, records from diierent sources, are cleansed and integrated in a centralized data store to be used for various purposes. Both, earlier and recent studies in data quality and record linkage, focus on various statistical models which make strong assumptions on the probabilities of attribute errors. In this study, we evaluate diierent models for record linkage that are built based on data only. We use a program that generates data with known error distributions and we also train classiication models, which we use to estimate the accuracy and the completeness of the record linking process. The results, indicate that the automated learning techniques are adequate for this process and that both their accuracy and their completeness is comparable to the accuracy and the completeness of other, mostly manual, processes. 1. Introduction The enterprise data is created, used and shared by a corporation in conducting business. This is a critical business asset that must be designed, analyzed, and managed with data quality as a guiding principle and not as an afterthought 17]. Poor data quality, that results from missing customer information, wrong address information, etc., undermines the customer satisfaction, leads to high and unnecessary costs and more importantly impacts decision making 21, 20, 16]. A number of reasons is responsible for bad data quality including: (a) multiple sources of data, (b) incompatible data, (c) data from multiple level of granularity, (d) redundant data, (e) corrupted and noisy data, and (f) missing attribute values. Data quality is achieved in three stages: the rst one is based on the cleansing, or scrubbing of data, the second on the matching or the linking of the records and the third one on the consolidation or the integration of the cleansed information with other internal or external sources. In the rst stage, the data is parsed, corrected, standardized and enhanced …